Hindi-English Language Identification, Named Entity Recognition and Back Transliteration: Shared Task System Description

This paper presents an algorithm for word level language identification, named entity recognition and classification, and transliteration of Indian language words written in the Roman script to their native Devanagari script from bilingual textual data. We propose the construction of an extensive, hierarchical structured dictionary and hierarchical rule-based classifier to expedite word search and language identification. The proposed method uses lexical, contextual and special character features particular to Hindi and English. With a few modifications to the system, the present solution can be replicated for other languages. The system we have submitted shows the best performance in English token level precision (0.895) and the second best in Indian language token recall (0.915). The transliteration level f-measure is relatively low (0.15); this can be significantly improved with a more representative and exhaustive training data.